Resolving multipath interference using a mixed active depth system

ABSTRACT

Aspects of the present disclosure relate to depth sensing using a device. An example device includes a light projector configured to project light in a first and a second distribution. The first and the second distribution include a flood projection when the device operates in a first mode and a pattern projection when the device operates in a second mode, respectively. The example device includes a receiver configured to detect reflections of light projected by the light projector. The example device includes a processor connected to a memory storing instructions. The processor is configured to determine first depth information based on reflections detected by the receiver when the device operates in the first mode, determine second depth information based on reflections detected by the receiver when the device operates in the second mode, and resolve multipath interference (MPI) using the first depth information and the second depth information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application claims priority to U.S. Provisional PatentApplication No. 62/857,804 entitled “MIXED ACTIVE DEPTH” and filed onJun. 5, 2019, which is assigned to the assignee hereof. The disclosuresof all prior applications are considered part of and are incorporated byreference in this patent application.

TECHNICAL FIELD

This disclosure relates generally to depth sensing systems andspecifically to improving the speed and accuracy at which active depthsystems generate depth information.

BACKGROUND

A device may determine distances of its surroundings using variousactive-or-passive depth sensing techniques. Passive depth sensingsystems measure reflected ambient light. Active depth sensing systemsemit waves from a non-ambient source of light (e.g., an illuminator, alight emitting diode (LED), a laser, or another suitable light source)and measure the corresponding reflected energy. For example, an activedepth sensing system, or device, may generate depth information,illustrating or otherwise indicating depths from the device to a sceneor one or more objects in the scene. The active depth sensing system mayemit one or more light pulses and measure reflections of the lightpulses from the objects or scene. Example active depth techniquesinclude time-of-flight (ToF) and structured light (SL).

In a ToF system, light is emitted from a transmitter (or “projector”),and a reflection of the light is received at a receiver (or “sensor”).The round-trip time of the light from the transmitter to the receiver isdetermined, and the distance or depth of an object reflecting theemitted light can be determined from the round trip time. ToF systemsare useful for many applications based on size, accuracy, performanceindices, and cost. For example, ToF systems provide depth informationfaster and more reliably at the hardware level than, for example, stereomulti-camera systems that require more complex processing.

A SL system may include a light emitter (or “transmitter” or“projector”) to project a distribution of infrared (IR) light (such as adistribution of IR light points) onto a scene. The system or device mayalso include a sensor (or “receiver”) that senses the reflections of thedistribution of light to determine distances of objects in the scene.The emitter and the sensor are separated by a distance, and displacementand distortion of the spatial distribution occurs at the sensor as aresult. The SL system determines a distance or depth of an objectreflecting the emitted light back to the system, for example, usingtriangulation with the displacement and distortion of the spatialdistribution and the distance between the transmitter and receiver.

SUMMARY

This Summary is provided to introduce in a simplified form a selectionof concepts that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tolimit the scope of the claimed subject matter.

Some aspects of the present disclosure relate to a device for depthsensing. An example device includes a light projector, a receiver, amemory storing instructions, and a processor connected to the memory. Anexample light projector is configured to project light in a firstdistribution including a flood projection when the device operates in afirst mode. The example light projector is further configured to projectlight in a second distribution including a pattern projection when thedevice operates in a second mode. An example receiver is configured todetect reflections of light projected by the light projector. An exampleprocessor is configured to determine first depth information based onreflections detected by the receiver when the device operates in thefirst mode. The example processor is further configured to determinesecond depth information based on reflections detected by the receiverwhen the device operates in the second mode. The example processor isfurther configured to resolve multipath interference (MPI) using thefirst depth information and the second depth information.

Some other aspects of the present disclosure relate to a method fordepth sensing using a device. An example method includes projectinglight in a first distribution including a flood projection when thedevice operates in a first mode. The example method further includesprojecting light in a second distribution including a pattern projectionwhen the device operates in a second mode. The example method furtherincludes detecting reflections of light projected by the lightprojector. The example method further includes determining first depthinformation based on reflections detected by the receiver when thedevice operates in the first mode. The example method further includesdetermining second depth information based on reflections detected bythe receiver when the device operates in the second mode. The examplemethod further includes resolving multipath interference (MPI) using thefirst depth information and the second depth information.

Some other aspects of the present disclosure relate to a non-transitorycomputer-readable medium storing instructions that, when executed by oneor more processors of an apparatus, causes the apparatus to performoperations. Example operations include projecting light in a firstdistribution including a flood projection when the device operates in afirst mode. The example operations further include projecting light in asecond distribution including a pattern projection when the deviceoperates in a second mode. The example operations further includedetecting reflections of light projected by the light projector. Theexample operations further include determining first depth informationbased on reflections detected by the receiver when the device operatesin the first mode. The example operations further include determiningsecond depth information based on reflections detected by the receiverwhen the device operates in the second mode. The example operationsfurther include resolving multipath interference (MPI) using the firstdepth information and the second depth information.

Some other aspects of the present disclosure relate to a device. Anexample device includes means for projecting light in a firstdistribution including a flood projection when the device operates in afirst mode. The example device further includes means for projectinglight in a second distribution including a pattern projection when thedevice operates in a second mode. The example device further includesmeans for detecting reflections of light projected by the lightprojector. The example device further includes means for determiningfirst depth information based on reflections detected by the receiverwhen the device operates in the first mode. The example device furtherincludes means for determining second depth information based onreflections detected by the receiver when the device operates in thesecond mode. The example device further includes means for resolvingmultipath interference (MPI) using the first depth information and thesecond depth information.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the present disclosure are illustrated by way of example, andnot by way of limitation, in the figures of the accompanying drawingsand in which like reference numerals refer to similar elements.

FIG. 1 shows an example time-of-flight (ToF) system.

FIG. 2A shows an example ToF system including an emitter and a sensor.

FIG. 2B shows an example sensor pixel.

FIG. 2C shows a pulse diagram illustrating a pulsed signal from atransmitter and a corresponding reflection signal received at a sensor.

FIG. 3 shows an example ToF system.

FIG. 4A shows an example environment in which multipath interference(MPI) may affect ToF depth sensing.

FIG. 4B shows another example environment in which MPI may affect ToFdepth sensing.

FIG. 4C shows another example environment in which MPI may affect ToFdepth sensing.

FIG. 5A shows an example scene including a corner of a room with an apexpoint where two of the walls intersect with the ceiling.

FIG. 5B shows an example depth map of the scene from FIG. 5A asgenerated by a conventional ToF system.

FIG. 6 shows a graph of the corresponding X-distance and Z-distance forthe scene of FIG. 5A.

FIG. 7 shows an example structured light (SL) system.

FIG. 8 shows a block diagram of an example device including a mixed ToFand SL system.

FIG. 9 shows a timing diagram illustrating example operation timings formultiple components of a ToF and SL system.

FIG. 10A shows an example ToF and SL system operating in a first mode.

FIG. 10B shows the example ToF and SL system of FIG. 10A operating in asecond mode.

FIG. 11A is a simplified illustration of the ToF and SL system shown inFIG. 10A.

FIG. 11B is a simplified illustration of the ToF and SL system shown inFIG. 10B.

FIG. 12A shows an example switchable diffuser.

FIG. 12B shows another example switchable diffuser.

FIG. 13A shows a top-down view of an example sensor capable of operatingin a ToF mode and a SL mode.

FIG. 13B shows a top-down view of another example sensor capable ofoperating in a ToF mode and a SL mode.

FIG. 14 shows an example electrical circuit diagram for a demodulationpixel cell.

FIG. 15A shows an example electrical circuit diagram for a globalshutter (GS) pixel array.

FIG. 15B shows another example electrical circuit diagram for a GS pixelarray.

FIG. 16A shows an example electrical circuit diagram for a hybrid ToFand SL pixel array operating in a GS implementation.

FIG. 16B shows another example electrical circuit diagram for a hybridToF and SL pixel array operating in a GS implementation.

FIG. 17 shows an example electrical circuit diagram for a hybrid ToF andSL pixel array operating in a rolling shutter (RS) implementation.

FIG. 18 shows an example timing diagram illustrating a hybrid ToF and SLpixel array of a device with a RS implementation operating in a SL mode.

FIG. 19 shows an example timing diagram illustrating a hybrid ToF and SLpixel array of a device with a RS implementation operating in a ToFmode.

FIG. 20 shows a flowchart illustrating an example process for depthsensing according to some implementations.

FIG. 21A shows a flowchart illustrating an example process for depthsensing according to some implementations.

FIG. 21B shows a flowchart illustrating an example process for depthsensing according to some implementations.

FIG. 21C shows a flowchart illustrating an example process for depthsensing according to some implementations.

FIG. 21D shows a flowchart illustrating an example process for depthsensing according to some implementations.

DETAILED DESCRIPTION

Aspects of the present disclosure relate to active depth systems andinclude a system that incorporates time-of-flight (ToF) and structuredlight (SL) techniques into a single device. ToF systems generallyproduce higher resolution depth information than SL systems. However,the depth information generated by conventional ToF systems oftensuffers from image artifacts induced by multipath interference (MPI). Onthe other hand, depth information generated by SL systems is relativelyunaffected by MPI.

In some aspects of the present disclosure, an active depth sensingsystem may combine the advantages of ToF (e.g., higher resolution) andSL (e.g., no MPI). For example, the present embodiments disclose anactive depth sensing system configured to operate in a mixed ToF and SLmode. The system may determine SL information for an object or scene andutilize the SL information to mitigate the effects of MPI whendetermining ToF depth information for the object or scene. The systemmay include a composite sensor, a hybrid emitter, and/or a programmablearchitecture, thus saving device space and requiring fewer devicecomponents than a conventional ToF system while also improving the speedand accuracy at which the system generates depth information.

In the following description, numerous specific details are set forth,such as examples of specific components, circuits, and processes toprovide a thorough understanding of the present disclosure. The term“coupled” as used herein means connected directly to or connectedthrough one or more intervening components or circuits. Also, in thefollowing description and for purposes of explanation, specificnomenclature is set forth to provide a thorough understanding of thepresent disclosure. However, it will be apparent to one skilled in theart that these specific details may not be required to practice theteachings disclosed herein. In other instances, well-known circuits anddevices are shown in block diagram form to avoid obscuring teachings ofthe present disclosure. Some portions of the detailed descriptions whichfollow are presented in terms of procedures, logic blocks, processes,and other symbolic representations of operations on data bits within acomputer memory. In the present disclosure, a procedure, logic block,process, or the like, is conceived to be a self-consistent sequence ofsteps or instructions leading to a desired result. The steps are thoserequiring physical manipulations of physical quantities. Usually,although not necessarily, these quantities take the form of electricalor magnetic signals capable of being stored, transferred, combined,compared, and otherwise manipulated in a computer system.

It should be borne in mind, however, that all of these and similar termsare to be associated with the appropriate physical quantities and aremerely convenient labels applied to these quantities. Unlessspecifically stated otherwise as apparent from the followingdiscussions, it is appreciated that throughout the present application,discussions utilizing the terms such as “accessing,” “receiving,”“sending,” “using,” “selecting,” “determining,” “normalizing,”“multiplying,” “averaging,” “monitoring,” “comparing,” “applying,”“updating,” “measuring,” “deriving,” “settling,” or the like, refer tothe actions and processes of a computer system, or similar electroniccomputing device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system registersand memories into other data similarly represented as physicalquantities within the computer system memories or registers or othersuch information storage, transmission or display devices.

In the figures, a single block may be described as performing a functionor functions; however, in actual practice, the function or functionsperformed by that block may be performed in a single component or acrossmultiple components, and/or may be performed using hardware, usingsoftware, or using a combination of hardware and software. To clearlyillustrate this interchangeability of hardware and software, variousillustrative components, blocks, modules, circuits, and steps aredescribed below generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem. Skilled artisans may implement the described functionality invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the present disclosure. Also, the example devices may includecomponents other than those shown, including well-known components suchas a processor, memory and the like.

Aspects of the present disclosure are applicable to any suitableelectronic device (such as security systems, smartphones, tablets,laptop computers, vehicles, drones, or other devices) including orcoupled to one or more active depth sensing systems. While describedbelow with respect to a device having or coupled to one light projector,aspects of the present disclosure are applicable to devices having anynumber of light projectors, and are therefore not limited to specificdevices.

The term “device” is not limited to one or a specific number of physicalobjects (such as one smartphone, one controller, one processing system,and so on). As used herein, a device may be any electronic device withone or more parts that may implement at least some portions of thisdisclosure. While the below description and examples use the term“device” to describe various aspects of this disclosure, the term“device” is not limited to a specific configuration, type, or number ofobjects. Additionally, the term “system” is not limited to multiplecomponents or specific embodiments. For example, a system may beimplemented on one or more printed circuit boards or other substrates,and may have movable or static components. While the below descriptionand examples use the term “system” to describe various aspects of thisdisclosure, the term “system” is not limited to a specificconfiguration, type, or number of objects.

FIG. 1 shows an example ToF system 100. The ToF system 100 may be usedto generate depth information of a scene including a surface 106, or maybe used for other applications for ranging surfaces or other portions ofthe scene. The ToF system 100 may include a transmitter 102 and areceiver 108. The transmitter 102 may be referred to as a “lightprojector,” “transmitter,” “projector,” “emitter,” and so on, and shouldnot be limited to a specific transmission component. Similarly, thereceiver 108 may be referred to as a “light sensor,” “detector,”“sensor,” “sensing element,” “photodetector,” and so on, and should notbe limited to a specific receiving component.

The transmitter 102 may be configured to transmit, emit, or projectsignals (such as a field of light) onto the scene. While ToF systems aredescribed in the examples as emitting light (which may includenear-infrared (NIR)), signals at other frequencies may be used, such asmicrowaves, radio frequency signals, sound, and so on. The presentdisclosure should not be limited to a specific range of frequencies forthe emitted signals.

The transmitter 102 transmits light 104 toward a scene including asurface 106. The transmitted light 104 includes light pulses 114 atknown time intervals (such as at a regular time interval). The receiver108 includes a sensor 110 to sense the reflections 112 of thetransmitted light 104. The reflections 112 include the reflected lightpulses 116, and the ToF system 100 determines a round trip time 122 forthe light by comparing the timing 118 of the transmitted light pulses tothe timing 120 of the reflected light pulses 116. The distance of thesurface 106 from the ToF system 100 may be calculated to be half theround trip time multiplied by the speed of the emissions (such as thespeed of light for light emissions).

The sensor 110 may include an array of photodiodes to measure or sensethe reflections. Alternatively, the sensor 110 may include a CMOS sensoror other suitable photo-sensitive sensor including a number of pixels orregions for sensing. The ToF system 100 identifies the reflected lightpulses 116 as sensed by the sensor 110 when the magnitudes of the pulsesare greater than a value. For example, the ToF system 100 measures amagnitude of the ambient light and other interference without the signaland determines if further measurements are greater than the previousmeasurement by a value. FIG. 2A shows an example ToF system 220including an emitter 225 and a sensor 230. The emitter 225 transmitslight pulses toward an object 235, and the light pulses are reflectedback by the object 235 toward the sensor 230. The light reflected backto the sensor 230 may have a different phase than the light emitted fromthe emitter 225. FIG. 2B shows an example sensor pixel 240 that may beincluded, for example, in the sensor 230. The sensor pixel 240 includesa photodiode 242 for converting photons from the reflected light intoelectrical current. The sensor pixel 240 may also include one or morecapacitors (e.g., C1 and C2) to store energy from the current. The ToFsystem 220 may calculate a distance between the ToF system 220 and theobject 235 in part by comparing the voltages (e.g., V0 and V180) withtheir corresponding phases (e.g., Φ0 and Φ180, respectively).

FIG. 2C shows a pulse diagram 260 illustrating a pulsed signal emittedfrom the emitter 225 and a corresponding reflection of the pulsed signal(e.g., reflected by an object, such as the object 235 of FIG. 2A)received at the sensor 230. The reflected signal is phase-delayedrelative to the pulsed signal. The ToF system 220 may open and close ashutter to expose the sensor 230 at a number of particular phase offsets(e.g., Phase 0°, Phase 180°, Phase 90°, and Phase 270°) relative to thepulsed signal. During each exposure cycle (shaded in gray), electricalcharge may be stored by one or more storage elements, such as bycapacitors C1 and C2 of FIG. 2B. For example, during a first exposurecycle, C1 may store a charge (Q1) and C2 may store a charge (Q2), whereQ1 is the accumulated charge from the reflected signal when the shutteris open at the 0° phase offset, and where Q2 is the accumulated chargefrom the reflected signal when the shutter is open at the 180° phaseoffset. During a second exposure cycle, C1 may store a charge (Q3) andC2 may store a charge (Q4), where Q3 is the accumulated charge from thereflected signal when the shutter is open at the 90° phase offset, andwhere Q4 is the accumulated charge from the reflected signal when theshutter is open at the 270° phase offset. The ToF system 220 cancalculate the phase offset (φ) between the pulsed signal and thereflected signal based on the charges stored across C1 and C2 for eachof the exposure cycles:

$\varphi = {\tan^{- 1}\frac{{Q3} - {Q4}}{{Q1} - {Q2}}}$

The calculated phase offset φ between the pulsed signal and thereflected signal is proportional to the distance d between thecorresponding sensor pixel (such as the sensor pixel 240) and the object235:

$d = {\left( \frac{c}{2f} \right)*\left( \frac{\varphi}{2\pi} \right)}$where c is the speed of light and f is the frequency of the modulatedsignal. Based on the determined distances from each pixel of the sensor230 to the object 235, the ToF system 220 may generate depth informationfor the object in the scene.

FIG. 3 shows an example ToF system 300. The ToF system 300 includes anemitter 315 and a sensor 335. The emitter 315 pulses a signal 320 towardan object 325, and the sensor 335 receives a corresponding reflectedsignal 330. The pulsed signal 320 may be a modulated continuous-wave(AMCW) pulse of light. As described in connection with FIG. 2C, the ToFsystem 300 may determine a distance d from one or more pixels of thesensor 335 based on a proportional correlation to the phase difference φbetween the pulsed signal 320 and the reflected signal 330. The pulsedsignal 320 and the reflected signal 330 may follow a direct path 340.

Some environments (e.g., with corners, convex areas, and/or reflectivesurfaces) may cause different pulses of light to arrive at the ToFsystem sensor along multiple reflection paths and recombine at thesensor, which is known as MPI. For purposes of discussion herein, MPImay also be referred to as “multipath effects” or “MPI effects.” MPI maycause the ToF system to overestimate the amount of charge beingaccumulated for one or more phase offsets of the corresponding pulsedsignal. The overestimation may cause the ToF system to inaccuratelycalculate the corresponding phase shift φ between the pulsed andreflected signals. Thus, the ToF system may inaccurately calculate thecorresponding distance d from one or more of the sensor pixels to theobject or scene, which may cause distortions (or “bumps”) incorresponding depth information.

FIG. 4A shows an example environment 400 in which MPI may affect ToFdepth sensing. The ToF system includes an emitter 415 and a sensor 435.The scene includes an object 425 and an object 427. The object 427 mayhave a mirror-like surface. The emitter 415 transmits a pulsed signal420 and a pulsed signal 422 toward the object 425. The sensor 435receives corresponding reflected signal 430 and reflected signal 432,respectively. Similar to the pulsed signal 320 and the reflected signal330 of FIG. 3 , the pulsed signal 420 and the reflected signal 430follow a direct path 440. In contrast, the pulsed signal 422 and thereflected signal 432 follow an indirect path 450 (e.g., reflecting offof the object 427) such that the reflected signal 432 may arrive at thesensor 435 at the same time as the reflected signal 430, causing MPI.

FIG. 4B shows another example environment 460 in which MPI may affectToF depth sensing. The ToF system includes an emitter 445 and a sensor465. The scene includes an object 455 and an object 457. The object 457may have a semi-transparent surface. The emitter 445 transmits a pulsedsignal 421 and a pulsed signal 423. The sensor 465 receivescorresponding reflected signal 431 and reflected signal 433,respectively. The pulsed signal 421 and the reflected signal 431 followpath 441 (e.g., reflecting off of the object 455). The pulsed signal 423and the reflected signal 433 follow path 451 (e.g., reflecting off ofthe object 457). The reflected signal 433 may arrive at the sensor 465at the same time as the reflected signal 431, causing MPI.

FIG. 4C shows another example environment 470 in which MPI may affectToF depth sensing. The ToF system includes an emitter 475 and a sensor495. The scene includes an object 485 and an object 487, which mayrepresent two walls that intersect at a corner point. The emitter 475transmits a pulsed signal 491 toward the object 485, and the sensor 495receives the corresponding reflected signal 492. In addition, possiblydue to the reflective properties of the object 485 or the object 487,reflected signal 493 and reflected signal 494 may arrive at the sensor495 at the same time as the reflected signal 492, causing MPI.

FIG. 5A shows an example scene 500 including a corner of a room with anapex point 510 where two of the walls intersect with the ceiling. Itwill be understood that the apex point 510 is the furthest point (e.g.,the point having the highest z-distance) from the sensor of the ToFsystem. FIG. 5B shows an example depth map 550 of the scene from FIG. 5Aas generated by a conventional ToF system (“regular ToF”). However,similar to the example environment 470 of FIG. 4C, multiple reflectedsignals may arrive at the sensor (e.g., due to the corner of the roomwith apex point 560), causing MPI. A conventional ToF system maysuperimpose the multiple reflected signals, ultimately resulting in thecorresponding region in the depth map 550 appearing to have a uniformdepth.

To illustrate, FIG. 6 shows a graph 600 of a corresponding X-distance asmeasured by a ToF system and a corresponding Z-distance as measured by aToF system for the scene 500 of FIG. 5A. The x-distance may represent ahorizontal distance (in meters, m) from the center of the scene at 0 m.The z-distance may represent a depth distance from the sensor of the ToFsystem to an object in the scene, such as a surface of a wall. Thebottom plot represents the actual (“true”) x-distance and z-distance forthe corner, while the top plot represents the distance measured by theToF system. The true distance plot accurately shows the apex point ofthe corner of the room, which comes to a sharp point. The measureddistance plot depicts the apex point as a bowl-like curve. Thisinaccurate measurement yields incorrect distance calculations, such asshown in the depth map 550 of FIG. 5B.

FIG. 7 shows an example SL system 700. As described above, SL isrelatively unaffected by MPI and generally produces lower (more sparse)resolution depth information than ToF systems (e.g., the ToF system 100of FIG. 1 ). A SL system may transmit light in a distribution of points(or another suitable shape of focused light). For purposes of discussionherein, the distribution of points may be referred to as a “pattern,” a“SL pattern,” a “dot pattern,” or the like, and the pattern may bepredefined or random. The points of light may be projected on to ascene, and the reflections of the points of light may be received by theSL system. Depths of objects in a scene may be determined by comparingthe pattern of the received light and the pattern of the transmittedlight. In comparing the patterns, a portion of the predefineddistribution for the transmitted light may be identified in the receivedlight. A SL system may project a distribution of light (such as adistribution of light points or other shapes) using a structured lightprojector.

The light distribution emitted by a SL projector may not change. Denserdistributions of light (such as additional light points or moreinstances of focused light in an area than for sparser distributions oflight) may result in a higher resolution of depth information or agreater number of depths that may be determined. However, the intensityof individual light points are lower for denser distributions than forsparser distributions where the overall intensity is similar between thedistribution. As a result, interference may cause identifyingreflections of a denser distribution of light to be more difficult thanfor sparser distributions of light. For example, a SL projector mayproject IR light (such as NIR light) with a 905 nm or 940 nm wavelength(or other suitable wavelength). A SL receiver may receive reflections ofthe IR light as well as sunlight and other ambient light. Ambient lightmay cause interference of the IR light points. As a result, brightly litscenes (such as outdoor scenes in daylight) may cause more interferencethan darker scenes (such as indoor scenes or nighttime scenes) becauseof the additional ambient light being captured by the SL receiver.

The SL system 700 (which herein may also be called a SL system) may beused to generate depth information for a scene 706. For example, thescene 706 may include a face, and the SL system 700 may be used foridentifying or authenticating the face. The SL system 700 may include atransmitter 702 and a receiver 708. The transmitter 702 may be referredto as a “transmitter,” “projector,” “emitter,” and so on, and should notbe limited to a specific transmission component. Throughout thefollowing disclosure, the terms projector and transmitter may be usedinterchangeably. The receiver 708 may be referred to as a “detector,”“sensor,” “sensing element,” “photodetector,” and so on, and should notbe limited to a specific receiving component.

While the disclosure refers to the distribution as a light distribution,any suitable signals at other frequencies may be used (such as radiofrequency waves, sound waves, etc.). Further, while the disclosurerefers to the distribution as including a plurality of light points, thelight may be focused into any suitable size and dimensions. For example,the light may be projected in lines, squares, or any other suitabledimension. In addition, the disclosure may refer to the distribution asa codeword distribution, where a defined portion of the distribution(such as a predefined patch of light points) is referred to as acodeword. If the distribution of the light points is known, thecodewords of the distribution may be known. However, the distributionmay be organized in any way, and the present disclosure should not belimited to a specific type of distribution or type of signal or pulse.

The transmitter 702 may be configured to project or transmit adistribution 704 of light points onto the scene 706. The white circlesin the distribution 704 may indicate where no light is projected for apossible point location, and the black circles in the distribution 704may indicate where light is projected for a possible point location. Insome example implementations, the transmitter 702 may include one ormore light sources 724 (such as one or more lasers), a lens 726, and alight modulator 728. The transmitter 702 may also include an aperture722 from which the transmitted light escapes the transmitter 702. Insome implementations, the transmitter 702 may further include adiffractive optical element (DOE) to diffract the emissions from one ormore light sources 724 into additional emissions. In some aspects, thelight modulator 728 (to adjust the intensity of the emission) mayinclude a DOE. In projecting the distribution 704 of light points ontothe scene 706, the transmitter 702 may transmit one or more lasers fromthe light source 724 through the lens 726 (and/or through a DOE or lightmodulator 728) and onto the scene 706. The transmitter 702 may bepositioned on the same reference plane as the receiver 708, and thetransmitter 702 and the receiver 708 may be separated by a distancecalled the baseline (712).

In some example implementations, the light projected by the transmitter702 may be IR light. IR light may include portions of the visible lightspectrum and/or portions of the light spectrum that is not visible tothe naked eye. In one example, IR light may include NIR light, which mayor may not include light within the visible light spectrum, and/or IRlight (such as far infrared (FIR) light) which is outside the visiblelight spectrum. The term IR light should not be limited to light havinga specific wavelength in or near the wavelength range of IR light.Further, IR light is provided as an example emission from thetransmitter. In the following description, other suitable wavelengths oflight may be used. For example, light in portions of the visible lightspectrum outside the IR light wavelength range or ultraviolet light.Alternatively, other signals with different wavelengths may be used,such as microwaves, radio frequency signals, and other suitable signals.

The scene 706 may include objects at different depths from the SL system(such as from the transmitter 702 and the receiver 708). For example,objects 706A and 706B in the scene 706 may be at different depths. Thereceiver 708 may be configured to receive, from the scene 706,reflections 710 of the transmitted distribution 704 of light points. Toreceive the reflections 710, the receiver 708 may capture an image. Whencapturing the image, the receiver 708 may receive the reflections 710,as well as (i) other reflections of the distribution 704 of light pointsfrom other portions of the scene 706 at different depths and (ii)ambient light. Noise may also exist in the captured image.

In some example implementations, the receiver 708 may include a lens 730to focus or direct the received light (including the reflections 710from the objects 706A and 706B) on to the sensor 732 of the receiver708. The receiver 708 may also include an aperture 720. Assuming for theexample that only the reflections 710 are received, depths of theobjects 706A and 706B may be determined based on the baseline 712,displacement and distortion of the light distribution 704 (such as incodewords) in the reflections 710, and intensities of the reflections710. For example, the distance 734 along the sensor 732 from location716 to the center 714 may be used in determining a depth of the object706B in the scene 706. Similarly, the distance 736 along the sensor 732from location 718 to the center 714 may be used in determining a depthof the object 706A in the scene 706. The distance along the sensor 732may be measured in terms of number of pixels of the sensor 732 or adistance (such as millimeters).

In some example implementations, the sensor 732 may include an array ofphotodiodes (such as avalanche photodiodes) for capturing an image. Tocapture the image, each photodiode in the array may capture the lightthat hits the photodiode and may provide a value indicating theintensity of the light (a capture value). The image therefore may be thecapture values provided by the array of photodiodes.

In addition or alternative to the sensor 732 including an array ofphotodiodes, the sensor 732 may include a complementary metal-oxidesemiconductor (CMOS) sensor. To capture the image by a photosensitiveCMOS sensor, each pixel of the sensor may capture the light that hitsthe pixel and may provide a value indicating the intensity of the light.In some example implementations, an array of photodiodes may be coupledto the CMOS sensor. In this manner, the electrical impulses generated bythe array of photodiodes may trigger the corresponding pixels of theCMOS sensor to provide capture values.

The sensor 732 may include at least a number of pixels equal to thenumber of possible light points in the distribution 704. For example,the array of photodiodes or the CMOS sensor may include a number ofphotodiodes or a number of pixels, respectively, corresponding to thenumber of possible light points in the distribution 704. The sensor 732logically may be divided into groups of pixels or photodiodes (such as4×4 groups) that correspond to a size of a bit of a codeword. The groupof pixels or photodiodes may also be referred to as a bit, and theportion of the captured image from a bit of the sensor 732 may also bereferred to as a bit. In some example implementations, the sensor 732may include the same number of bits as the distribution 704.

If the light source 724 transmits IR light (such as NIR light at awavelength of, e.g., 940 nm), the sensor 732 may be an IR sensor toreceive the reflections of the NIR light. The sensor 732 may also beconfigured to capture an image using a flood illuminator (notillustrated). As illustrated, the distance 734 (corresponding to thereflections 710 from the object 706B) is less than the distance 736(corresponding to the reflections 710 from the object 706A). Usingtriangulation based on the baseline 712 and the distances 734 and 736,the differing depths of objects 706A and 706B in the scene 706 may bedetermined in generating depth information for the scene 706.Determining the depths may further include determining a displacement ora distortion of the distribution 704 in the reflections 710.

Although multiple separate components are illustrated in FIG. 7 , one ormore of the components may be implemented together or include additionalfunctionality. All described components may not be required for a SLsystem 700, or the functionality of components may be separated intoseparate components. Additional components not illustrated may alsoexist. For example, the receiver 708 may include a bandpass filter toallow signals having a determined range of wavelengths to pass onto thesensor 732 (thus filtering out signals with a wavelength outside of therange). In this manner, some incidental signals (such as ambient light)may be prevented from interfering with the captures by the sensor 732.The range of the bandpass filter may be centered at the transmissionwavelength for the transmitter 702. For example, if the transmitter 702is configured to transmit NIR light with a wavelength of 940 nm, thereceiver 708 may include a bandpass filter configured to allow NIR lighthaving wavelengths within a range of, e.g., 920 nm to 960 nm. Therefore,the examples described regarding FIG. 7 are for illustrative purposes,and the present disclosure should not be limited to the example SLsystem 700.

For a light projector (such as the transmitter 702), the light sourcemay be any suitable light source. In some example implementations, thelight source 724 may include one or more distributed feedback (DFB)lasers. In some other example implementations, the light source 724 mayinclude one or more vertical-cavity surface-emitting lasers (VCSELs).

A DOE is a material situated in the projection path of the light fromthe light source. The DOE may be configured to split a light point intomultiple light points. For example, the material of the DOE may be atranslucent or a transparent polymer with a known refractive index. Thesurface of the DOE may include peaks and valleys (varying the depth ofthe DOE) so that a light point splits into multiple light points whenthe light passes through the DOE. For example, the DOE may be configuredto receive one or more lights points from one or more lasers and projectan intended distribution with a greater number of light points thanemitted by the one or more lasers. While the Figures may illustrate thedepth of a DOE changing along only one axis of the DOE, the Figures areonly to assist in describing aspects of the disclosure. The peaks andvalleys of the surface of the DOE may be located at any portion of thesurface of the DOE and cause any suitable change in the depth ofportions of the DOE, and the present disclosure should not be limited toa specific surface configuration for a DOE.

If the light source 724 includes an array of lasers (such as a VCSELarray), a portion of the distribution of light points may be projectedby the array. A DOE may be used to replicate the portion in projectingthe distribution of light points. For example, the DOE may split theprojection from the array into multiple instances, and a pattern of theprojection may be a repetition of the projection from the array. In someexample implementations, the DOE may be configured to repeat theprojection vertically, horizontally, or at an angle between vertical andhorizontal relative to the projection. The repeated instances may beoverlapping, non-overlapping, or any suitable configuration. While theexamples describe a DOE configured to split the projection from thearray and stack the instances above and below one another, the presentdisclosure should not be limited to a specific type of DOE configurationand repetition of the projection.

FIG. 8 shows a block diagram of an example device 800 configured foractive depth sensing using ToF and SL techniques. It will be understoodthat ToF and SL are example active depth techniques and that the device800 may use other active depth techniques in some implementations. Insome embodiments, the device 800 may be configured to generatehigh-resolution depth information using ToF techniques while using SLtechniques to mitigate the effects of MPI in the depth information. Thedevice 800 may include or be coupled to an emitter 801, a sensor 802, aprocessor 804, a memory 806 storing instructions 808, and an activedepth controller 810 (which may include one or more signal processors812). The emitter 801 may include or be coupled to a DOE 805. The DOE805 may optionally be included in or coupled to the device 800. Theemitter 801 may include or be coupled to a diffuser 807. The diffuser807 may optionally be included in or coupled to the device 800. Forpurposes of discussion herein, the device 800 may be referred to as a“ToF and SL system.” Further for purposes of discussion herein, the “ToFand SL system” may instead refer to just one or more components of thedevice 800 (e.g., the active depth controller 810, the emitter 801, thesensor 802, the DOE 805, and/or the diffuser 807) and/or any othercomponents that may be used for active depth sensing. In someimplementations, the device may be a wireless communication device.

In some embodiments, the emitter 801 may be a single, hybrid laserprojector capable of switching between projecting a first distributionof light (e.g., with use of the diffuser 807) during a first projectionmode (e.g., a ToF projection mode) of the emitter 801 and projecting asecond distribution of light (e.g., with use of the DOE 805) during asecond projection mode (e.g., a SL projection mode) of the emitter 801.When operating in the SL projection mode, the DOE 805 may enable theemitter 801 to transmit the second distribution of light, which may be,for example, a known DOE dot pattern, a codeword DOE projection, arandom dot projection or distribution, or the like. The diffuser 807 maybe switchable such that the diffuser is “off” (or “disabled” or“switched off”) when the device 800 operates in the SL projection modeand is “on” (or “enabled” or “switched on”) when the device 800 operatesin the ToF projection mode. More specifically, when operating in the ToFprojection mode, the diffuser 807 is switched on, which causes theemitter 801 to transmit the second distribution of light (e.g., flooddistribution). Accordingly, the emitter 801 may be synchronized toproject a second distribution of light (e.g., a DOE distribution) duringthe SL projection mode and a second distribution of light (e.g., a fullflood frame) during a ToF projection mode. In some embodiments, theemitter 801 may include multiple projectors.

In some embodiments, the sensor 802 may be a single, hybrid ToF and SLsensor for receiving reflected light according to ToF and SL sensing (or“readout”) modes. The sensor 802 may be configured to switch betweenoperating in a first sensing mode (e.g., a ToF sensing mode) and asecond sensing mode (e.g., a SL sensing mode). For example, the sensor802 may be a composite CMOS image sensor configured to switch betweenoperating in (or alternating between) the ToF and SL sensing modes. Thesensing mode may depend on which distribution (e.g., DOE or flood) theemitter 801 is projecting. In some aspects, the sensor 802 may be basedon a monolithic pixel array architecture, for example, withTime-Division Multiplexed Read (TDMR) capabilities. In otherembodiments, the sensor 802 may include one or more generic ToF sensorsoperating in conjunction with multiple projectors.

In some embodiments, the active depth controller 810 may be acomputation element for calculating depth information. The active depthcontroller 810 may be configured to alternate between computing depthinformation using ToF techniques and computing depth information usingSL techniques. For purposes of discussion herein, depth informationcalculated using SL techniques may also be referred to as “SL depthinformation,” “SL information,” or the like. Similarly, for purposes ofdiscussion herein, depth information calculated using ToF techniques mayalso be referred to as “ToF depth information,” “ToF information,” orthe like. In some aspects, the active depth controller 810 may use SLdepth information as reference for calculating or supplementing ToFdepth information, which may help compensate for MPI errors in the ToFdepth information. In some embodiments, the sensor 802 may be aReconfigurable Instruction Cell Array (RICA), which is a proprietary,real-time, low-power, (re)programmable, image signal processing (ISP),active sensing, processing engine. In some aspects, stacking the RICAprogrammable implementation with the hybrid NIR sensor described hereinmay enable the active depth controller 810 to switch programming“on-the-fly” to toggle computing SL depth information and ToF depthinformation while reducing a number of components for a sensor (e.g.,the sensor 802). In other embodiments, the active depth controller 810may be a generic sensor.

In some aspects, the active depth controller 810 may be configured tocontrol (or otherwise operate) at least one of the emitter 801 and thesensor 802 to synchronize their respective operating modes, such thatthe emitter 801 and the sensor 802 concurrently operate in either theirrespective SL modes or ToF modes. In some aspects, the active depthcontroller 810 may be controlled, work in conjunction with, or otherwisebe operated by one or more other components of the device 800, such asthe processor 804 and/or the memory 806.

The device 800 may optionally include or be coupled to a display 814 anda number of input/output (I/O) components 816. The sensor 802 may be, orotherwise may be coupled to, a camera, such as a single camera, a dualcamera module, or a module with any number of other camera sensors (notpictured). The signal processor 812 may be configured to processcaptures from the sensor 802. The device 800 may further include one ormore optional sensors 820 (such as a gyroscope, magnetometer, inertialsensor, NIR sensor, and so on) coupled to the processor 804. The device800 may also include a power supply 818, which may be coupled to orintegrated into the device 800. The device 800 may include additionalfeatures or components not shown.

The memory 806 may be a non-transient or non-transitory computerreadable medium storing computer-executable instructions 808 to performall or a portion of one or more operations described in this disclosure.The processor 804 may be one or more suitable processors capable ofexecuting scripts or instructions of one or more software programs (suchas instructions 808) stored within the memory 806. In some aspects, theprocessor 804 may be one or more general purpose processors that executeinstructions 808 to cause the device 800 to perform any number offunctions or operations. In additional or alternative aspects, theprocessor 804 may include integrated circuits or other hardware toperform functions or operations without the use of software. While shownto be coupled to each other via the processor 804 in the example of FIG.8 , the processor 804, the memory 806, the active depth controller 810,the optional display 814, the optional I/O components 816, and theoptional sensors 820 may be coupled to one another in variousarrangements. For example, the processor 804, the memory 806, the activedepth controller 810, the optional display 814, the optional I/Ocomponents 816, and/or the optional sensors 820 may be coupled to eachother via one or more local buses (not shown for simplicity).

The display 814 may be any suitable display or screen allowing for userinteraction and/or to present items (such as depth information or apreview image of the scene) for viewing by a user. In some aspects, thedisplay 814 may be a touch-sensitive display. The I/O components 816 maybe or include any suitable mechanism, interface, or device to receiveinput (such as commands) from the user and to provide output to theuser. For example, the I/O components 816 may include (but are notlimited to) a graphical user interface, keyboard, mouse, microphone andspeakers, squeezable bezel or border of the device 800, physical buttonslocated on device 800, and so on. The display 814 and/or the I/Ocomponents 816 may provide a preview image or depth information for thescene to a user and/or receive a user input for adjusting one or moresettings of the device 800 (such as adjusting an intensity of emissionsby emitter 801, determining or switching one or more operating modes ofthe device 800, adjusting a field of emission of the emitter 801, and soon).

The active depth controller 810 may include, or may otherwise be coupledto, a signal processor 812, which may be one or more processors toprocess captures from the sensor 802. The active depth controller 810may be configured to switch at least one of the emitter 801 and thesensor 802 between one or more operating modes. The active depthcontroller 810 may alternatively or additionally include a combinationof specific hardware and the ability to execute software instructions.

The emitter 801 may vary its field of emission for different operatingmodes. In some example implementations, the emitter 801 may include afocusing apparatus for adjusting the size of the field ofemission/transmission. In one example, mirrors attached to actuators(such as microelectromechanical systems (MEMS) actuators) may adjust thefocus of the light emissions from the emitter 801. In another example,an adjustable holographic optical element (HOE) may adjust the focus ofthe light emissions from the emitter 801. In a further example, aformable DOE (such as a piezoelectric material to adjust the shape) maybe adjusted to focus the diffracted points of light emitted.

In some other example implementations, the device 800 may emit lightusing a plurality of light emitters (not shown) instead of, or incombination with, the emitter 801. The emitters may include a firstgroup of light emitters (e.g., of a first array of light emitters) foremitting light with a first field of transmission. The emitters mayfurther include a second or different group of light emitters (e.g., ofa second array of light emitters) for emitting light with a second fieldof transmission. The first field may be larger than the second field ata common depth from the emitter 801. In some example implementations,the first group of light emitters may be active for a first mode of theemitter 801, and the second group of light emitters may be active for asecond mode of the emitter 801.

FIG. 9 shows a timing diagram 900 illustrating an example operation of aToF and SL system including a sensor 909, an emitter 949, and acontroller 979. The sensor 909, emitter 949, and controller 979 may beexample embodiments of the sensor 802, emitter 801, and active depthcontroller 810, respectively, of FIG. 8 . It will be understood that ToFand SL are example active depth techniques and that the system may useother active depth techniques in some implementations.

The example timing diagram 900 shows three projection cycles for theemitter 949: a first projection cycle ending at time 901, a secondprojection cycle ending at time 902, and a third projection cycle endingat time 903. The emitter 949 may project a first distribution of lightduring each of the projection cycles. The first distribution of lightmay be a flood distribution for a first projection mode, such as a ToFprojection mode. For example, the emitter 949 may project a flooddistribution 950, a flood distribution 952, and a flood distribution 954during a ToF projection mode for each of the first, second, and thirdprojection cycles, respectively. For purposes of discussion herein, aflood distribution may also be referred to as a “flood illumination” ora “diffuse light.” The emitter 949 may also project a seconddistribution of light during each of the projection cycles. The seconddistribution of light may be a DOE distribution for a second projectionmode, such as a SL projection mode. For example, the emitter 949 mayproject a DOE distribution 970, a DOE distribution 972, and a DOEdistribution 974 during the SL projection mode for each of the first,second, and third projection cycles, respectively. For purposes ofdiscussion herein, a DOE distribution may also be referred to as a “DOEpattern,” a “DOE projection,” a “SL distribution,” a “SL pattern,”and/or a “SL projection.”

The example timing diagram 900 shows three sensing cycles for the sensor909: a first sensing cycle ending at time 901, a second sensing cycleending at time 902, and a third sensing cycle ending at time 903. Thesensor 909 may read out two frames of ToF sensor data (during a ToFsensing mode) and one frame of SL sensor data (during a SL sensing mode)for each sensing cycle. The sensor 909 may be configured to operate inthe ToF sensing mode at the same time as when the emitter 949 isconfigured to operate in the ToF projection mode. The sensor 909 may beconfigured to operate in the SL sensing mode at the same time as whenthe emitter 949 is configured to operate in the SL projection mode.Similar to the system described in FIG. 2C, during the ToF sensing mode,the sensor 909 may be exposed at a number of particular phase offsets(e.g., Phase 0°, Phase 180°, Phase 90°, and Phase 270°) relative to apulsed signal from the emitter 949. The sensor 909 may accumulate andstore an amount of charge (Q) for each of the particular phase offsets.

For example, during a first exposure, the sensor 909 may read out thefirst frame of ToF sensor data 910 based on a Q1 and a Q2, where Q1 isthe charge accumulated at 0° phase offset, and where Q2 is the chargeaccumulated at 180° phase offset. During a second exposure, the sensor909 may read out a second frame of ToF sensor data 912 based on a Q3 anda Q4, where Q3 is the charge accumulated at 90° phase offset, and whereQ4 is the charge accumulated at 270° phase offset. Similarly, the sensor909 may read-out a first frame of ToF sensor data 914 and a second frameof ToF sensor data 916 during the second sensing cycle, and the sensor909 may read-out a first frame of ToF sensor data 918 and a second frameof ToF sensor data 920 during the third sensing cycle. The sensor 909may read-out a frame of SL sensor data 930 during the first sensingcycle, a frame of SL sensor data 932 during the second sensing cycle,and a frame of SL sensor data 934 during the third sensing cycle.

After each sensing cycle, the controller 979 may calculate SL depthinformation (Z(SL)) using SL sensor data. For example, the controller979 may calculate SL depth information (Z(SL)) 980, Z(SL) 982, and Z(SL)984 after each of the first, second, and third sensing cycle,respectively.

After calculating Z(SL), the controller 979 may calculate ToF depthinformation (Z(ToF+SL)) for the corresponding sensing cycle using ToFsensor data and Z(SL). For example, the controller 979 may calculate ToFdepth information Z(ToF+SL) 990, Z(ToF+SL) 992, and Z(ToF+SL) 994 aftercalculating each of Z(SL) 980, Z(SL) 982, and Z(SL) 984, respectively.In some aspects, the controller 979 may calculate Z(ToF+SL) using oneframe of ToF sensor data for the corresponding sensing cycle, or, sinceToF sensing techniques are susceptible to noise, the controller 979 maycalculate Z(ToF+SL) using more than one frame of ToF sensor data for thecorresponding sensing cycle. For example, to calculate Z(ToF+SL) 990,the controller 979 may average the first frame of ToF sensor data 910with the second frame of ToF sensor data 912. The controller 979 maycalculate Z(SL) and Z(ToF+SL) any time during the next sensing cycle.For example, the controller 979 may calculate Z(SL) 980 and Z(ToF+SL)990 between time 901 and time 902.

In the example of FIG. 9 , the system may generate high-resolution andhigh-accuracy depth information without (or with at least mitigated) MPIartifacts using the sparse depth information from the SL mode as abaseline to eliminate multipath effects from the ToF mode. In accordancewith the embodiments described herein, the system may generate the depthinformation without (or with at least mitigated) MPI artifacts using asingle sensor (e.g., the sensor 909), a single emitter (e.g., theemitter 949), and/or a single controller (e.g., the controller 979).

FIG. 10A shows an example ToF and SL system 1000 operating in a first(e.g., ToF) mode. The system 100 includes an emitter 1010 and a sensor1045. During the ToF mode, the emitter 1010 may operate in a ToFprojection mode, and the sensor 1045 may operate in a ToF sensing mode,as described with respect to FIG. 9 . The emitter 1010 and the sensor1045 may be example embodiments of the emitter 801 and the sensor 802,respectively, of FIG. 8 . The ToF and SL system 1000 includes a DOE 1020and a diffuser 1030 coupled to the front of the DOE 1020. In someaspects, the diffuser 1030 may be a switchable diffuser, such asdescribed above with respect to the diffuser 807 of FIG. 8 . Thediffuser 1030 may be switched on (e.g., for flood distribution) duringthe ToF projection mode. Specifically, during the ToF projection mode,the emitter 1010 may transmit pulsed signal 1011 toward an object 1040while the diffuser 1030 diffuses the emitted light to project a flooddistribution (e.g., a uniform illumination) onto the scene. A reflectedsignal 1041 may arrive at the sensor 1045, and the sensor 1045 maycalculate ToF depth information based on an amount of time for the lightto be reflected back to the sensor 1045 for each pixel.

FIG. 10B shows the example ToF and SL system 1000 operating in a second(e.g., SL) mode. During the SL mode, the emitter 1010 may operate in aSL projection mode, and the sensor 1045 may operate in a SL sensingmode, as described with respect to FIG. 9 . The diffuser 1030 may beswitched off (e.g., functioning as a transparent piece of glass) duringthe SL projection mode. Specifically, during the SL projection mode, theemitter 1010 may project a DOE distribution toward a scene, which willpass through the diffuser 1030 relatively unaffected (as projected light1013), and onto the scene (e.g., as a dot matrix pattern). Reflectedlight 1043 may arrive at the sensor 1045, and the sensor 1045 maycalculate SL depth information, Z(SL), based on how the projected light1013 distorts on the scene. An active depth controller (such as theactive depth controller 810 of FIG. 8 ) may use Z(SL) during acalculation of ToF depth information, Z(ToF+SL), as described withrespect to FIG. 9 . In this manner, the active depth controller 810 mayreduce or eliminate multipath artifacts in the ToF depth information.Moreover, the present embodiments may provide accurate high-resolutiondepth sensing using a single sensor (e.g., the sensor 1045), a singleemitter (e.g., the emitter 1010), and/or a single controller (e.g., theactive depth controller 810).

FIG. 11A is a simplified illustration of a ToF and SL system 1100operating in a first (e.g., ToF) mode. The ToF and SL system 1100 may bean example implementation of the ToF and SL system 1000, as describedwith respect to FIG. 10A. The laser 1110 emits light through the DOE1120. In the ToF mode, the diffuser 1130 may be switched on (indicatedwith solid gray), and the DOE distribution may be diffused when passingthrough the diffuser 1130, which floods the scene 1140 withillumination.

FIG. 11B is a simplified illustration of the ToF and SL system 1100 ofFIG. 11A operating in a second (e.g., SL) mode. The ToF and SL system1100 may be an example implementation of the ToF and SL system 1000, asdescribed with respect to FIG. 10B. The laser 1110 emits light through aDOE 1120. In the SL mode, the diffuser 1130 may be switched off(indicated with a dotted pattern), and the DOE distribution may beprojected through the diffuser 1130 (e.g., unaltered) and onto the scene1140.

FIG. 12A shows an example switchable diffuser 1260 operating in a first(e.g., ToF) mode, in accordance with some embodiments. The switchablediffuser 1260 may be an example embodiment of the diffuser 807 of FIG. 8. In some aspects, the switchable diffuser 1260 may be a liquid crystal(LC)-based switchable diffuser. The switchable diffuser 1260 has a layerof dielectric material (e.g., a LC layer 1264) having refractive index,

$n_{a} = {\frac{2\left( {n_{o} + n_{e}} \right)}{3}.}$A glass substrate may be positioned between a DOE substrate 1262 and theLC layer 1264. In some aspects, the DOE substrate 1262 has Indium tinoxide (ITO) coding on its backside and faces a diffusion layer 1268. Thediffusion layer 1268 may be fabricated in the dielectric material layerand have a refractive index, n_(o). A pair of conductive materials 1263with a voltage 1265 may control an orientation of the LC molecules inthe LC layer 1264. In the example of FIG. 12A, a voltage is not appliedacross the LC layer 1264 (e.g., the voltage 1265, V=0). Thus, therefractive index of the LC material 1264 remains n_(a), and the LCmolecules in the LC layer 1264 remain randomly oriented. Since, in thisexample, the refractive index of the LC layer 1264 and the diffusionlayer 1268 are different, the diffuser 1260 may be “switched on.” Thus,the light 1261 scatters (e.g., diffuses) at the junction between the LClayer 1264 and the diffusion layer 1268, which projects a flooddistribution for the ToF mode.

FIG. 12B shows the example switchable diffuser 1260 of FIG. 12Aoperating in a second (e.g., SL) mode, in accordance with someembodiments. The switchable diffuser 1260 may be an example embodimentof the diffuser 807 of FIG. 8 . In the example of FIG. 12B, a voltage,V₀, is applied across the LC layer 1264. Thus, the refractive index ofthe LC material 1264 becomes n_(o), and the LC molecules in the LC layer1264 move into alignment. Since, in this example, the refractive indexof the LC layer 1264 and the diffusion layer 1268 are the same, thediffuser 1260 may be “switched off.” Thus, the light 1261 goes straightthrough the junction between the LC layer 1264 and the diffusion layer1268 (e.g., relatively unaffected), which projects a DOE distributionfor the SL mode.

FIG. 13A shows a top-down view of an example sensor 1300 capable ofoperating in a first sensing mode (e.g., a ToF sensing mode) and asecond sensing mode (e.g., a SL sensing mode). The sensor 1300 may be anexample embodiment of the sensor 802 of FIG. 8 . The sensor 1300 mayalso be referred to herein as a composite, hybrid, NIR, pixel imagesensor or receiver and/or an integrated (photo)detector array. In someaspects, the sensor 1300 may have a monolithic CMOS architecture with aTDMR configuration for alternating between the ToF and SL sensing modes.In some other aspects, the sensor 1300 may have a charge-coupled device(CCD) architecture. The sensor 1300 may combine a Global Shutter (GS)Pixel 1310 with a ToF Pixel 1330. The sensor 1300 may be configured toswitch between a ToF sensing mode and a SL sensing mode at the same timethat an emitter (such as the emitter 801 of FIG. 8 ) switches between aToF projection mode and a SL projection mode, as described with respectto FIG. 9 .

The sensor 1300 may include one or more multi-phase, lock-in pixel cellsfor determining a phase difference between a pulsed signal and areceived signal for the ToF sensing mode. The sensor 1300 may alsoinclude a GS demodulation pixel cell for operating the sensor 1300 inthe SL sensing mode. For example, the sensor 1300 includes two total NIRgates (NIR gate 1320 and NIR gate 1322) for detecting NIR light fromreflected signals. The NIR gate 1320 may detect NIR light during the SLsensing mode. Since the GS pixel 1310 and the ToF pixel 1330 are notelectrically isolated, the NIR gate 1320 (in addition to the NIR gate1322) may also detect NIR light during the ToF sensing mode. The sensor1300 also has two ToF gates (ToF gate 1340 and ToF gate 1342) forreceiving reflected light pulses having a phase shift of φ relative tothe pulsed signal and converting the optical signals to electricalsignals. Each ToF gate is coupled to a voltage source (not shown) thatprovides time-varying control signals. Each of the two ToF gates arecoupled to a readout circuit (not shown) for reading out charges (Q)collected from the reflected light. The sensor 1300 is an example of atwo-phase sensor, since the first ToF gate 1340 may read out a firstcollected charge (e.g., Q1) at a first phase shift (e.g., 0°) relativeto the emitted light and the second ToF gate 1342 may read out a secondcollected charge (e.g., Q2) at a second phase shift (e.g., 180°)relative to the emitted light. Each readout circuitry may include somenumber of transistors, such as selection gates, source-followers, resetgates, select gates, or any other suitable circuitry.

In this manner, the sensor 1300 may operate as a single, two-phasesensor for a mixed-mode ToF and SL system and generate high-resolutionand high-accuracy depth information without (or with at least mitigated)MPI artifacts using the sparse depth information from the SL mode as abaseline to eliminate multipath effects from the ToF mode.

FIG. 13B shows a top-down view of an example sensor 1350 capable ofoperating in a first sensing mode (e.g., a ToF sensing mode) and asecond sensing mode (e.g., a SL sensing mode). The sensor 1350 may be anexample embodiment of the sensor 802 of FIG. 8 . The sensor 1350 mayalso be referred to herein as a composite, hybrid, NIR, pixel imagesensor or receiver and/or an integrated (photo)detector array. In someaspects, the sensor 1350 may have a monolithic CMOS architecture with aTDMR configuration for alternating between the ToF and SL sensing modes.In some other aspects, the sensor 1350 may have a CCD architecture. Thesensor 1350 may combine features from a GS Pixel 1360 and a ToF Pixel1380. The sensor 1350 may be configured to switch between a ToF sensingmode and a SL sensing mode at the same time that an emitter (such as theemitter 801 of FIG. 8 ) switches between a ToF projection mode and a SLprojection mode, as described with respect to FIG. 9 .

The sensor 1350 may include one or more multi-phase, lock-in pixel cellsfor determining a phase difference between a pulsed signal and areceived signal for the ToF sensing mode. The sensor 1350 may alsoinclude a GS demodulation pixel cell for operating the sensor 1350 inthe SL sensing mode. For example, the sensor 1350 includes a NIR gate1370, coupled to the GS pixel 1360, for detecting NIR light fromreflected signals. The NIR gate 1370 may detect NIR light during the SLsensing mode. The GS pixel 1360 and the ToF pixel 1380 are electricallyisolated. Thus, unlike the ToF pixel 1330 of FIG. 13A, the GS pixel 1360may not share the NIR gate 1370 with the ToF pixel 1380 during the ToFmode. Instead, the ToF pixel 1380 has four ToF gates (ToF gate 1390, ToFgate 1392, ToF gate 1394, and ToF gate 1396) for receiving reflectedlight pulses having a phase shift of φ relative to the pulsed signal andconverting the optical signals to electrical signals. Each ToF gate iscoupled to a voltage source (not shown) that provides time-varyingcontrol signals. Each of the four ToF gates are coupled to a readoutcircuit (not shown) for reading out charges (Q) collected from thereflected light. The sensor 1350 is an example of a four-phase sensor,since the first ToF gate 1390 may read out a first collected charge(e.g., Q1) at a first phase shift (e.g., 0°) relative to the emittedlight, the second ToF gate 1392 may read out a second collected charge(e.g., Q2) at a second phase shift (e.g., 180°) relative to the emittedlight, the third ToF gate 1394 may read out a third collected charge(e.g., Q3) at a third phase shift (e.g., 90°) relative to the emittedlight, and the fourth ToF gate 1396 may read out a fourth collectedcharge (e.g., Q4) at a fourth phase shift (e.g., 270°) relative to theemitted light. Each readout circuitry may include some number oftransistors, such as selection gates, source-followers, reset gates,select gates, or any other suitable circuitry.

In this manner, the sensor 1350 may operate as a single, four-phasesensor for a mixed-mode ToF and SL system and generate high-resolutionand high-accuracy depth information without (or with at least mitigated)MPI artifacts using the sparse depth information from the SL mode as abaseline to eliminate multipath effects from the ToF mode.

FIG. 14 shows an example electrical circuit diagram for a demodulationpixel cell 1400. The demodulation pixel cell 1400 may include aphotodiode 1420 coupled to a ground potential 1410. The photodiode 1420may convert light (e.g., photons) from reflected signals to electricalcurrent, which flows to a transistor 1430 and a transistor 1460 coupledin parallel to the photodiode 1420. The transistor 1430 and thetransistor 1460 may block the current from flowing to a capacitor (C1)and a capacitor (C2), respectively. C1 may be coupled to a groundpotential 1450, and C2 may be coupled to a ground potential 1480. Insome aspects, at least one of the transistor 1430 or the transistor 1460may be field-effect transistors (FETs). In some aspects, at least one ofthe transistor 1430 and the transistor 1460 may bemetal-oxide-semiconductor field-effect transistors (MOSFETs).

The demodulation pixel cell 1400 may capture ToF sensor data which maybe used for generating ToF depth information. For example, during afirst exposure cycle, C1 may store a first charge (Q1) from a reflectedsignal when the shutter is opened at a first phase offset (e.g., Φ₁=0°)relative to the transmitted signal, and C2 may store a second charge(Q2) from the reflected signal when the shutter is opened at a secondphase offset (e.g., Φ₂=180°) relative to the transmitted signal. Duringa second exposure cycle, C1 may store a third charge (Q3) from thereflected signal when the shutter is opened at a third phase offset(e.g., Φ₁=90°) relative to the transmitted signal, and C2 may store afourth charge (Q4) from the reflected signal when the shutter is openedat a fourth phase offset (e.g., Φ₂=270°) relative to the transmittedsignal. The phase offset, φ, between the transmitted signal and thereflected signal may be calculated based on the charges stored across C1and C2 for each of the exposure cycles, which allows calculation ofcorresponding ToF depth information:

${D = {\frac{c*\Delta t}{2} = {\frac{c*\theta}{2\omega} = \frac{c*\varphi}{4\pi*f_{mod}}}}}{\sigma_{depth} \equiv {\frac{1}{\sqrt{2}*SNR}*\frac{1}{2\pi}*\frac{c}{2*f_{mod}}}}$$\varphi = {{\tan^{- 1}\left( \frac{V_{90} - V_{270}}{V_{0} - V_{180}} \right)} = {\frac{\pi}{2}*\left( {1 - \frac{V_{0} - V_{180}}{{{V_{0} - V_{180}}} + {{V_{90} - V_{270}}}}} \right)}}$where D represents depth information, c is the speed of light

$\left( {{i.e.},{3*10^{8}\frac{m}{\sec}}} \right),$f_(mod) represents the modulation frequency of the transmitted signal,V₀−V₁₈₀ represents the integrated electrical signals for Φ₁ and Φ₂during the first exposure cycle, V₉₀−V₂₇₀ represents the integratedelectrical signals for Φ₁ and Φ₂ during the second exposure cycle, andσ_(depth) represents a depth accuracy.

FIG. 15A shows an example electrical circuit diagram for a GS pixelarray 1500. The GS pixel array 1500 may also be referred to herein as anNIR GS imager. The GS pixel array 1500 includes two shared GSphotodiodes, PD1 and PD2. Each of PD1 and PD2 may absorb photons (e.g.,from light reflected back from a scene and/or an object) during a SLsensing mode. Each of PD1 and PD2 is coupled to a floating storagediode, SD1 and SD2, respectively. SD1 and SD2 may operate as storagenode elements for charge accumulation and readout from the photodiodesPD1 and PD2. Each of the storage diodes SD1 and the SD2 is coupled to atransfer gate, TG1 and TG2, respectively. TG1 and TG2 may be transistorswith relatively low voltage drops. Charge from PD1 and PD2 may flow to atransistor LOD1 and a transistor LOD2, respectively, which are eachcoupled to a supply voltage, Vddpix 1565.

The GS pixel array 1500 includes a capacitor FD for accumulating charge.The capacitor FD is coupled to a transistor TS1, which is coupled to,for example, the storage diode SD1. The capacitor FD is also coupled toa transistor TS2, which is coupled to, for example, the storage diodeSD2. The capacitor FD is further coupled to a reset switch, RST. WhenRST is closed, charge may flow to Vddpix 1575. When RST is open, chargemay flow to a source follower amplifier, SF_AMP. Because the sourcevoltage of SF_AMP remains proportional to the gate voltage, SF_AMP mayconvert charge to voltage and toggle a Select switch, SEL. When SEL isopen (e.g., during a SL mode, when TS1 and TS2 are open), Vddpix 1575may be isolated, and a relatively small amount of charge from each of aseries of signal pulses (e.g., a pulse train) may accumulate across thecapacitor FD. When SEL is closed (e.g., after each of the SL modes, whenTS1 and TS2 are closed), the series of accumulated signal pulses may betransferred from each of TG1 and TG2 to an output terminal, Vout. Voutmay be coupled to a current source, I_bias. Accordingly, the GS pixelarray 1500 may capture SL sensor data for generating SL depthinformation.

FIG. 15B shows an example electrical circuit diagram for a GS pixelarray 1550. The GS pixel array 1550 may also be referred to herein as anNIR GS imager. The GS pixel array 1500 includes two shared GSphotodiodes, PD1 and PD2. Each of PD1 and PD2 may absorb photons (e.g.,from light reflected back from a scene and/or an object) during a SLsensing mode. Each of PD1 and PD2 is coupled to a CCD-readout memory,MEM1 and MEM2, respectively. MEM1 and MEM2 may operate as storage nodeelements for charge accumulation and readout from the photodiodes PD1and PD2. Each of MEM1 and MEM2 is coupled to a transfer gate, TG1 andTG2, respectively. TG1 and TG2 may be transistors with relatively lowvoltage drops. Charge from PD1 and PD2 may flow to a transistor LOD1 anda transistor LOD2, respectively, which are each coupled to a supplyvoltage, Vddpix 1568.

The GS pixel array 1550 includes a capacitor FD for accumulating charge.The capacitor FD is coupled to a transistor TS1, which is coupled to,for example, the CCD-readout memory, MEM1. The capacitor FD is alsocoupled to a transistor TS2, which is coupled to, for example, theCCD-readout memory, MEM2. The capacitor FD is further coupled to a resetswitch, RST. When RST is closed, charge may flow to Vddpix 1578. WhenRST is open, charge may flow to a source follower amplifier, SF_AMP.Because the source voltage of SF_AMP remains proportional to the gatevoltage, SF_AMP may convert charge to voltage and toggle a Selectswitch, SEL. When SEL is open (e.g., during a SL mode, when TS1 and TS2are open), Vddpix 1578 may be isolated, and a relatively small amount ofcharge from each of a series of signal pulses (e.g., a pulse train) mayaccumulate across the capacitor FD. When SEL is closed (e.g., after eachof the SL modes, when TS1 and TS2 are closed), the series of accumulatedsignal pulses may be transferred from each of TG1 and TG2 to an outputterminal, Vout. Vout may be coupled to a current source, I_bias.Accordingly, the GS pixel array 1550 may capture SL sensor data forgenerating SL depth information.

FIG. 16A shows an example electrical circuit diagram for a GS pixelarray 1600. The pixel array 1600 may also be referred to herein as ahybrid NIR GS imager and may be capable of operating in a ToF sensingmode and a SL sensing mode. The GS pixel array 1600 may be an exampleembodiment of the sensor 802 of FIG. 8 . The GS pixel array 1600includes two shared GS photodiodes, PD1 and PD2. Each of PD1 and PD2 mayabsorb photons (e.g., from light reflected back from a scene and/or anobject) during a SL sensing mode. PD1 is coupled to two floating storagediodes, SD1 and SD2, which may operate as storage node elements forcharge accumulation and readout from PD1. PD2 is coupled to two floatingstorage diodes, SD3 and SD4, which may operate as storage node elementsfor charge accumulation and readout from PD2. Each of SD1-SD4 is coupledto a transfer gate, TG1-TG4, respectively. Each of TG1-TG4 may be atransistor with a relatively low voltage drop. Charge from PD1 may flowto a transistor LOD1, which is coupled to a supply voltage, Vddpix 1647.Charge from PD2 may flow to a transistor LOD2, which is coupled to asupply voltage, Vddpix 1657.

The GS pixel array 1600 includes capacitors FD1 and FD2 for accumulatingcharge from reflected signals. FD1 is coupled to transistors TS1 andTS3, which are coupled to SD1 and SD3, respectively. FD2 is coupled totransistors TS2 and TS4, which are coupled to SD2 and SD4, respectively.Each of FD1 and FD2 is coupled to a reset switch, RST1 and RS2,respectively. When either of RST1 and RS2 is closed, charge may flow toVddpix 1665 and Vddpix 1675, respectively. When either of RST1 and RST2is open, charge may flow to a source follower amplifier, SF_AMP1 andSF_AMP2, respectively. Because the source voltage of SF_AMP1 and SF_AMP2remains proportional to the gate voltage, SF_AMP1 and SF_AMP2 mayconvert charge to voltage and toggle a corresponding Select switch, SEL1and SEL2, respectively.

During the ToF sensing mode, each of TS2, TG2, TS3, TG3, TS1, TG1, TS4,and TG4 may be closed (activated) and the pixel array 1600 maydemodulate multiple phases of reflected signals. SEL1 and SEL2 may alsobe open during the ToF sensing mode, which may isolate Vddpix 1665 andVddpix 1675, allowing a relatively small amount of charge from each of aseries of signal pulses (e.g., a pulse train) to accumulate across FD1and FD2. When SEL1 and SEL2 are closed, the series of accumulated signalpulses may be transferred from TG1 and TG4 to output terminals, Vout_nand Vout_n+1, respectively. Vout_n may be coupled to a current source,I_bias1, and Vout_n+1 may be coupled to a current source, I_bias2.Accordingly, the GS pixel array 1600 may capture SL sensor data forgenerating SL depth information. During the ToF sensing mode, a laser ofan emitter (such as the emitter 801 of FIG. 8 ) may operate with arelatively narrow duty cycle and turn on for a relatively short amountof time for each pulse. Thus, eye safety may be increased and powerconsumption may be reduced as compared with a rolling shutter (RS) pixelarray (not pictured).

During the SL sensing mode, each of TS2, TG2, TS3, and TG3 (e.g., halfof the read-out circuitry) may be open, and each of TS1, TG1, TS4, andTG4 (e.g., the other half of the read-out circuitry) may be closed. Inthis manner, reflected signals may be captured in dual-phase (e.g., oneon the left, and one on the right) at different time frames. SEL1 andSEL2 may also be open during the SL sensing mode, which may isolateVddpix 1665 and Vddpix 1675, allowing a relatively small amount ofcharge from each of a series of signal pulses (e.g., a pulse train) toaccumulate across FD1 and FD2. When SEL1 and SEL2 are closed, the seriesof accumulated signal pulses may be transferred from TG1 and TG4 tooutput terminals, Vout_n and Vout_n+1, respectively. Vout_n may becoupled to a current source, I_bias1, and Vout_n+1 may be coupled to acurrent source, I_bias2. Accordingly, the GS pixel array 1600 maycapture SL sensor data for generating SL depth information.

In this manner, the pixel array 1600 may operate as a hybrid GS sensorfor a mixed-mode ToF and SL system for generating high-resolution andhigh-accuracy depth information without (or with at least mitigated) MPIartifacts using the sparse depth information from the SL mode as abaseline to eliminate multipath effects from the ToF mode.

FIG. 16B shows an example electrical circuit diagram for a GS pixelarray 1650. The pixel array 1650 may also be referred to herein as ahybrid NIR GS imager and may be capable of operating in a ToF sensingmode and a SL sensing mode. The GS pixel array 1650 may be an exampleembodiment of the sensor 802 of FIG. 8 . The GS pixel array 1650includes two shared GS photodiodes, PD1 and PD2. Each of PD1 and PD2 mayabsorb photons (e.g., from light reflected back from a scene and/or anobject) during a SL sensing mode. PD1 is coupled to two CCD-readoutmemories, MEM1 and MEM4, which may operate as storage node elements forcharge accumulation and readout from PD1. PD2 is coupled to twoCCD-readout memories, MEM2 and MEM3, which may operate as storage nodeelements for charge accumulation and readout from PD2. Each of MEM1-MEM4is coupled to a transfer gate, TG1-TG4, respectively. Each of TG1-TG4may be a transistor with a relatively low voltage drop. Charge from PD1may flow to a transistor LOD1, which is coupled to a supply voltage,Vddpix 1647. Charge from PD2 may flow to a transistor LOD2, which iscoupled to a supply voltage, Vddpix 1657.

The GS pixel array 1650 includes capacitors FD1 and FD2 for accumulatingcharge from reflected signals. FD1 is coupled to transistors TS1 andTS3, which are coupled to MEM1 and MEM3, respectively. FD2 is coupled totransistors TS2 and TS4, which are coupled to MEM2 and MEM4,respectively. Each of FD1 and FD2 is coupled to a reset switch, RST1 andRS2, respectively. When either of RST1 and RS2 is closed, charge mayflow to Vddpix 1665 and Vddpix 1675, respectively. When either of RST1and RST2 is open, charge may flow to a source follower amplifier,SF_AMP1 and SF_AMP2, respectively. Because the source voltage of SF_AMP1and SF_AMP2 remains proportional to the gate voltage, SF_AMP1 andSF_AMP2 may convert charge to voltage and toggle a corresponding Selectswitch, SEL1 and SEL2, respectively.

During the ToF sensing mode, each of TS2, TG2, TS3, TG3, TS1, TG1, TS4,and TG4 may be closed (activated) and the pixel array 1650 maydemodulate multiple phases of reflected signals. SEL1 and SEL2 may alsobe open during the ToF sensing mode, which may isolate Vddpix 1665 andVddpix 1675, allowing a relatively small amount of charge from each of aseries of signal pulses (e.g., a pulse train) to accumulate across FD1and FD2. When SEL1 and SEL2 are closed, the series of accumulated signalpulses may be transferred from TG1 and TG4 to output terminals, Vout_nand Vout_n+1, respectively. Vout_n may be coupled to a current source,I_bias1, and Vout_n+1 may be coupled to a current source, I_bias2.Accordingly, the GS pixel array 1650 may capture SL sensor data forgenerating SL depth information. During the ToF sensing mode, a laser ofan emitter (such as the emitter 801 of FIG. 8 ) may operate with arelatively narrow duty cycle and turn on for a relatively short amountof time for each pulse. Thus, eye safety may be increased and powerconsumption may be reduced as compared with a RS pixel array (notpictured).

During the SL sensing mode, each of TS2, TG2, TS3, and TG3 (e.g., halfof the read-out circuitry) may be open, and each of TS1, TG1, TS4, andTG4 (e.g., the other half of the read-out circuitry) may be closed. Inthis manner, reflected signals may be captured in dual-phase (e.g., oneon the left, and one on the right) at different time frames. SEL1 andSEL2 may also be open during the SL sensing mode, which may isolateVddpix 1665 and Vddpix 1675, allowing a relatively small amount ofcharge from each of a series of signal pulses (e.g., a pulse train) toaccumulate across FD1 and FD2. When SEL1 and SEL2 are closed, the seriesof accumulated signal pulses may be transferred from TG1 and TG4 tooutput terminals, Vout_n and Vout_n+1, respectively. Vout_n may becoupled to a current source, I_bias1, and Vout_n+1 may be coupled to acurrent source, I_bias2. Accordingly, the GS pixel array 1650 maycapture SL sensor data for generating SL depth information.

In this manner, the pixel array 1650 may operate as a hybrid GS sensorfor a mixed-mode ToF and SL system for generating high-resolution andhigh-accuracy depth information without (or with at least mitigated) MPIartifacts using the sparse depth information from the SL mode as abaseline to eliminate multipath effects from the ToF mode.

FIG. 17 shows an example electrical circuit diagram for a RS pixel array1700. The pixel array 1700 may also be referred to herein as a hybridNIR RS imager and may be capable of operating in a ToF sensing mode anda SL sensing mode. The RS pixel array 1700 may be an example embodimentof the sensor 802 of FIG. 8 . The pixel array 1700 may be configured toread-out signal line-by-line and thus, to operate in a constant wavemode (e.g., at a particular duty cycle) so as to expose each line of theRS for an equal amount of time. The RS pixel array 1700 includes fourshared RS photodiodes, PD1-PD4. Each of PD1-PD4 may absorb photons(e.g., from light reflected back from a scene and/or an object) during aSL sensing mode. PD1 is coupled to two transfer gates, TG1 and TG2. PD2is coupled to two transfer gates, TG3 and TG4. PD3 is coupled to twotransfer gates, TG5 and TG6. PD4 is coupled to two transfer gates, TG7and TG8. Each of TG1-TG8 may be a transistor with a relatively lowvoltage drop.

The RS pixel array 1700 includes capacitors FD1 and FD2 for accumulatingcharge from reflected signals. Each of FD1 and FD2 is coupled to a resetswitch, RST1 and RS2, respectively. When either of RST1 and RS2 isclosed, charge may flow to Vddpix 1765 and Vddpix 1775, respectively.When either of RST1 and RST2 is open, charge may flow to a sourcefollower amplifier, SF_AMP1 and SF_AMP2, respectively. Because thesource voltage of SF_AMP1 and SF_AMP2 remains proportional to the gatevoltage, SF_AMP1 and SF_AMP2 may convert charge to voltage and toggle acorresponding Select switch, SEL1 and SEL2, respectively.

During the ToF sensing mode, each of TG1-TG8 may be closed (activated)and the pixel array 1700 may demodulate multiple phases of reflectedsignals. SEL1 and SEL2 may be open during the ToF sensing mode, whichmay isolate Vddpix 1765 and Vddpix 1775, allowing a relatively smallamount of charge from each of a series of signal pulses (e.g., a pulsetrain) to accumulate across FD1 and FD2. When SEL1 is closed, a seriesof accumulated signal pulses may be transferred from TG1, TG3, TG5, andTG7 to output terminal, Vout1. When SEL2 is closed, a series ofaccumulated signal pulses may be transferred from TG2, TG4, TG6, and TG8to output terminal, Vout2. Vout1 may be coupled to a current source,I_bias1, and Vout2 may be coupled to a current source, I_bias2.Accordingly, the RS pixel array 1700 may capture ToF sensor data forgenerating ToF depth information.

During the SL sensing mode, each of TG1, TG4, TG5, and TG8 (e.g., halfof the read-out circuitry) may be closed, and each of TG2, TG3, TG6, andTG7 (e.g., the other half of the read-out circuitry) may be open. Inthis manner, reflected signals may be captured in dual-phase (e.g., oneon the left, and one on the right) at different time frames. SEL1 andSEL2 may also be open during the SL sensing mode, which may isolateVddpix 1765 and Vddpix 1775, allowing a relatively small amount ofcharge from each of a series of signal pulses (e.g., a pulse train) toaccumulate across FD1 and FD2.

In this manner, the pixel array 1700 may operate as a hybrid RS sensorfor a mixed-mode ToF and SL system for generating high-resolution andhigh-accuracy depth information without (or with at least mitigated) MPIartifacts using the sparse depth information from the SL mode as abaseline to eliminate multipath effects from the ToF mode.

FIG. 18 shows an example timing diagram 1800 depicting an RS sensoroperating in a SL mode. The RS sensor may be an example embodiment ofthe ToF and SL pixel array 1700 of FIG. 17 . With reference to FIG. 17 ,for example, the pixel array 1700 may utilize only half of the transfergates (TGs) (e.g., TG1, TG4, TG5, and TG8) when operating in the SLmode. The signal for TG1 and TG4 may activate during every other (e.g.,n) first horizontal sync (H_sync) period (e.g., which immediatelyfollows the vertical sync (V_sync) period) for each of the Reset andReadout periods during the SL mode. The signal for TG5 and TG8 mayactivate during the other (e.g., n+1) horizontal sync (H_sync) periodsfor each of the Reset and Readout periods during the SL mode. That is,each of TG2, TG3, TG6, and TG7 may remain deactivated while operating inthe SL mode. Activation timing for the RST and SEL switches during theSL mode are also shown in FIG. 18 . The integration time shows theperiod during which charge is accumulated at the RS sensor, as describedwith respect to FIG. 17 . The RS sensor may alternate reading out SLdata (e.g., SL depth information) and ToF data (e.g., ToF depthinformation).

FIG. 19 shows an example timing diagram 1900 depicting an RS sensoroperating in a ToF mode. The RS sensor may be an example embodiment ofthe ToF and SL pixel array 1700 of FIG. 17 . With reference to FIG. 17 ,for example, the pixel array 1700 may utilize all of the transfer gates(TGs) (e.g., TG1-TG8) when operating in the ToF mode. The signal forTG1, TG3, TG5, and TG7 may activate (while TG2, TG4, TG6, and TG8 aredeactivated) during every other (e.g., 2n+1) first horizontal sync(H_sync) period, which may be referred to herein as an out-of-phase rawsignal, or S₁₈₀. The signal for TG2, TG4, TG6, and TG8 may activate(while TG1, TG3, TG5, and TG7 are deactivated) during the other (e.g.,2n) horizontal sync (H_sync) periods, which may be referred to herein asan in-phase raw signal, or S₀. Activation timing for the RST and SELswitches during the SL mode are also shown in FIG. 19 . The integrationtime shows the period during which charge from the reflected signals isaccumulated at the RS sensor, as described with respect to FIG. 17 . Areadout operation period follows each integration period. During eachreadout period, the FD nodes (e.g., FD 1730 and FD 1767 of FIG. 17 )operate as storage nodes, and the TG gates are deactivated (or “turnedoff”), allowing for the sensor to extract the accumulated charge beforeresetting the FD nodes for the next exposure cycle. In an aspect, the RSsensor may compensate (or “cancel”) background light (e.g., ambientlight or noise) by capturing an additional frame while an emitter (suchas the emitter 801 of FIG. 8 ) is deactivated. Accordingly, the RSsensor may subtract a relatively dark onset signal from the phases ofraw signals (e.g., S₀ and S₁₈₀) when generating depth information.

FIG. 20 shows a flowchart illustrating an example process 2000 for depthsensing according to some implementations. The process 2000 may beperformed by a device such as the device 800 described above withreference to FIG. 8 . In some implementations, the process 2000 beginsin block 2002 with projecting light in a first distribution including aflood projection when the device operates in a first mode. In block2004, the process 2000 proceeds with projecting light in a seconddistribution including a pattern projection when the device operates ina second mode. In block 2006, the process 2000 proceeds with detectingreflections of light projected by the light projector. In block 2008,the process 2000 proceeds with determining first depth information basedon reflections detected by the receiver when the device operates in thefirst mode. In block 2010, the process 2000 proceeds with determiningsecond depth information based on reflections detected by the receiverwhen the device operates in the second mode. In block 2012, the process2000 proceeds with resolving MPI using the first depth information andthe second depth information. In some implementations, the patternprojection may be created by a DOE disposed between the switchablediffuser and a light source of a light projector. In someimplementations, the device may be a wireless communication device.

FIG. 21A shows a flowchart illustrating an example process 2110 fordepth sensing according to some implementations. The process 2110 may beperformed by a device such as the device 800 described above withreference to FIG. 8 . In some implementations, the process 2110 may bean example of the processes of block 2008 and block 2010 of FIG. 20 fordetermining the first depth information and determining the second depthinformation, respectively. The process 2110 begins in block 2112 withdetermining the first depth information using ToF techniques. In block2114, the process 2110 proceeds with determining the second depthinformation using SL techniques.

FIG. 21B shows a flowchart illustrating an example process 2120 fordepth sensing according to some implementations. The process 2120 may beperformed by a device such as the device 800 described above withreference to FIG. 8 . In some implementations, the process 2120 may bean example of the processes of block 2002 and block 2004 of FIG. 20 forprojecting light, and begins in block 2122 with activating a switchablediffuser to project the flood projection. In block 2124, the process2120 proceeds with deactivating the switchable diffuser to project thepattern projection.

In some implementations, the light projector may be a birefringentmaterial disposed between the DOE and the switchable diffuser forapplying a voltage across a refractive material. The switchable diffusermay have a first refractive index and the birefringent material may be aliquid crystal material having a second refractive index. In someaspects, the first refractive index and the second refractive index maybe different when the light projector does not apply the voltage acrossthe refractive material. In some aspects, the first refractive index andthe second refractive index may be the same when the light projectordoes apply the voltage across the refractive material. In some aspects,the light projector may not apply the voltage across the refractivematerial when the device operates in the first mode. In some aspects,the light projector may apply the voltage across the refractive materialwhen the device operates in the second mode.

FIG. 21C shows a flowchart illustrating an example process 2130 fordepth sensing according to some implementations. The process 2130 may beperformed by a device such as the device 800 described above withreference to FIG. 8 . In some implementations, the process 2130 may bean example of the process of block 2006 of FIG. 20 for detectingreflections of light projected by the light projector, and begins inblock 2132 with detecting reflections of the first distribution when thedevice operates in the first mode. In block 2134, the process 2130proceeds with detecting reflections of the second distribution when thedevice operates in the second mode. In block 2136, the process 2130proceeds with determining at least one phase difference between theprojected light and the detected reflections when the device operates inthe first mode.

In some implementations, a monolithic pixel sensor including at leastone GS pixel cell and at least one lock-in pixel cell may determine theat least one phase difference in block 2136. In some aspects, thereceiver may detect, via the at least one GS pixel cell, NIR light basedon the reflections detected by the receiver when the device operates inthe second mode. In some aspects, the lock-in pixel cell may include twoToF gates for operating in a two-phase mode. In some aspects, the atleast one GS pixel cell and the at least one lock-in pixel cell may beisolated and the lock-in pixel cell may include four ToF gates foroperating in a four-phase mode.

FIG. 21D shows a flowchart illustrating an example process 2140 fordepth sensing according to some implementations. The process 2140 may beperformed by a device such as the device 800 described above withreference to FIG. 8 . The process 2140 may alternate between block 2142(detecting reflections of the first distribution) and block 2144(detecting reflections of the second distribution using TDMR). In someimplementations, block 2142 may be an example of block 2132 of FIG. 21Cfor detecting reflections of the first distribution. In someimplementations, block 2144 may be an example of block 2134 of FIG. 21Cfor detecting reflections of the second distribution. In someimplementations, a CMOS device and a CCD may alternate the detecting inprocess 2140.

The techniques described herein may be implemented in hardware,software, firmware, or any combination thereof, unless specificallydescribed as being implemented in a specific manner. Any featuresdescribed as modules or components may also be implemented together inan integrated logic device or separately as discrete but interoperablelogic devices. If implemented in software, the techniques may berealized at least in part by a non-transitory processor-readable storagemedium (such as the memory 806 in the device 800 of FIG. 8 ) includinginstructions 808 that, when executed by the processor 804 (or the activedepth controller 810), cause the device 800 to perform one or more ofthe methods described above. The non-transitory processor-readable datastorage medium may form part of a computer program product, which mayinclude packaging materials.

The non-transitory processor-readable storage medium may include randomaccess memory (RAM) such as synchronous dynamic random access memory(SDRAM), read only memory (ROM), non-volatile random access memory(NVRAM), electrically erasable programmable read-only memory (EEPROM),FLASH memory, other known storage media, and the like. The techniquesadditionally, or alternatively, may be realized at least in part by aprocessor-readable communication medium that carries or communicatescode in the form of instructions or data structures and that can beaccessed, read, and/or executed by a computer or other processor.

The various illustrative logical blocks, modules, circuits andinstructions described in connection with the embodiments disclosedherein may be executed by one or more processors, such as the processor804 or the active depth controller 810 in the device 800 of FIG. 8 .Such processor(s) may include but are not limited to one or more digitalsignal processors (DSPs), general purpose microprocessors, applicationspecific integrated circuits (ASICs), application specific instructionset processors (ASIPs), field programmable gate arrays (FPGAs), or otherequivalent integrated or discrete logic circuitry. The term “processor,”as used herein may refer to any of the foregoing structures or any otherstructure suitable for implementation of the techniques describedherein. In addition, in some aspects, the functionality described hereinmay be provided within dedicated software modules or hardware modulesconfigured as described herein. Also, the techniques could be fullyimplemented in one or more circuits or logic elements. A general purposeprocessor may be a microprocessor, but in the alternative, the processormay be any conventional processor, controller, microcontroller, or statemachine. A processor may also be implemented as a combination ofcomputing devices, e.g., a combination of a DSP and a microprocessor, aplurality of microprocessors, one or more microprocessors in conjunctionwith a DSP core, or any other such configuration.

While the present disclosure shows illustrative aspects, it should benoted that various changes and modifications could be made hereinwithout departing from the scope of the appended claims. For example,while the projectors are illustrated as including a lens to direct lighttoward a diffractive element, a projector may not include a lens or mayinclude multiple lenses. In another example, the electricity applied bythe device or light projector in adjusting the projection may bealternating current (AC) or direct current (DC), and the voltage may beconstant or non-constant. The electricity therefore may be any suitableelectricity for adjusting the projection. Additionally, the functions,steps or actions of the method claims in accordance with aspectsdescribed herein need not be performed in any particular order unlessexpressly stated otherwise. For example, the steps of the describedexample operations, if performed by the device 800, the active depthcontroller 810, the processor 804, and/or the memory 806, may beperformed in any order and at any frequency. Furthermore, althoughelements may be described or claimed in the singular, the plural iscontemplated unless limitation to the singular is explicitly stated.Accordingly, the disclosure is not limited to the illustrated examplesand any means for performing the functionality described herein areincluded in aspects of the disclosure.

The invention claimed is:
 1. A device, comprising: a light projectorconfigured to: project first light onto a scene in a first distributionincluding a flood projection when the device operates in a first mode;and project second light onto the scene in a second distributionincluding a pattern projection when the device operates in a secondmode; a receiver configured to detect reflections of light projected bythe light projector; a memory storing instructions; and a processor,connected to the memory, configured to: determine first depthinformation based on an amount of time for the first projected light tobe reflected back to the receiver when the device operates in the firstmode; determine second depth information based on distortion of at leastone pattern of the second projected light by one or more objects in ascene as detected by the receiver when the device operates in the secondmode; and compensate for multipath interference (MPI) errors in thefirst depth information at least in part by using the second depthinformation as a reference for removing the MPI errors from the firstdepth information.
 2. The device of claim 1, wherein the processor isfurther configured to: determine the first depth information usingtime-of-flight (ToF) techniques; and determine the second depthinformation using structured light (SL) techniques.
 3. The device ofclaim 1, wherein the light projector comprises a switchable diffuser andis further configured to: activate the switchable diffuser to projectthe flood projection.
 4. The device of claim 3, wherein the patternprojection is created by a diffractive optical element (DOE) disposedbetween the switchable diffuser and a light source of the lightprojector.
 5. The device of claim 4, wherein the light projectorcomprises a birefringent material disposed between the DOE and theswitchable diffuser, and wherein deactivating the switchable diffusercomprises: applying a voltage across a refractive material.
 6. Thedevice of claim 5, wherein the switchable diffuser has a firstrefractive index and the birefringent material is a liquid crystalmaterial having a second refractive index, wherein the first refractiveindex and the second refractive index are different when the lightprojector does not apply the voltage across the refractive material, andwherein the first refractive index and the second refractive index arethe same when the light projector does apply the voltage across therefractive material.
 7. The device of claim 6, wherein the lightprojector does not apply the voltage across the refractive material whenthe device operates in the first mode, and wherein the light projectordoes apply the voltage across the refractive material when the deviceoperates in the second mode.
 8. The device of claim 3, wherein the lightprojector is further configured to: deactivate the switchable diffuserto project the pattern projection.
 9. The device of claim 1, wherein thereceiver is further configured to: detect reflections of the firstdistribution when the device operates in the first mode; and detectreflections of the second distribution when the device operates in thesecond mode.
 10. The device of claim 1, wherein to the processor isfurther configured to: generate a depth map using the second depthinformation and the first depth information with at least mitigated MPI.11. The device of claim 1, wherein the receiver comprises a monolithicpixel sensor including at least one global shutter (GS) pixel cell andat least one lock-in pixel cell, and wherein the processor is furtherconfigured to: determine at least one phase difference between the firstprojected light and reflections detected by the monolithic pixel sensorwhen the device operates in the first mode.
 12. The device of claim 11,wherein the receiver is further configured to: detect, via the at leastone GS pixel cell, near infrared (NIR) light based on the reflectionsdetected by the receiver when the device operates in the second mode.13. The device of claim 11, wherein the lock-in pixel cell includes twotime-of-flight (ToF) gates for operating in a two-phase mode.
 14. Thedevice of claim 11, wherein the at least one GS pixel cell iselectrically isolated from the at least one lock-in pixel cell, andwherein the lock-in pixel cell includes four time-of-flight (ToF) gatesfor operating in a four-phase mode.
 15. The device of claim 1, whereinthe receiver comprises at least one of a complementary metal-oxidesemiconductor (CMOS) device and a charge-coupled device (CCD), andwherein the receiver is further configured to: alternate betweendetecting reflections of the first distribution and detectingreflections of the second distribution using time-division multiplexedread (TDMR).
 16. The device of claim 1, wherein the device is a wirelesscommunication device.
 17. A method for depth sensing using a device,comprising: projecting first light onto a scene in a first distributionincluding a flood projection when the device operates in a first mode;projecting second light onto the scene in a second distributionincluding a pattern projection when the device operates in a secondmode; detecting, by a receiver, reflections of the first projected lightand the second projected light; determining first depth informationbased on an amount of time for the first projected light to be reflectedback to by the receiver when the device operates in the first mode;determining second depth information based on distortion of at least onepattern of the second projected light by one or more objects in a sceneas detected by the receiver when the device operates in the second mode;and compensating for multipath interference (MPI) errors in the firstdepth information at least in part by using the second depth informationas a reference for removing the MPI errors from the first depthinformation.
 18. The method of claim 17, further comprising: determiningthe first depth information using time-of-flight (ToF) techniques; anddetermining the second depth information using structured light (SL)techniques.
 19. The method of claim 17, wherein the device includes aswitchable diffuser, and the method further comprises: activating theswitchable diffuser to project the flood projection.
 20. The method ofclaim 19, further comprising: deactivating the switchable diffuser toproject the pattern projection.
 21. The method of claim 17, furthercomprising: detecting reflections of the first distribution when thedevice operates in the first mode; and detecting reflections of thesecond distribution when the device operates in the second mode.
 22. Themethod of claim 21, further comprising: determining at least one phasedifference between the projected first light and reflections detected bythe receiver when the device operates in the first mode.
 23. The methodof claim 17, further comprising: generating a depth map using the seconddepth information and the first depth information with at leastmitigated MPI.
 24. The method of claim 17, further comprising:alternating between detecting reflections of the first distribution anddetecting reflections of the second distribution using time-divisionmultiplexed read (TDMR).
 25. The method of claim 17, wherein the deviceis a wireless communication device.
 26. A non-transitorycomputer-readable medium storing instructions that, when executed by oneor more processors of an apparatus, causes the apparatus to performoperations comprising: projecting first light onto a scene in a firstdistribution including a flood projection when the apparatus operates ina first mode; projecting second light onto the scene in a seconddistribution including a pattern projection when the apparatus operatesin a second mode; detecting reflections of the first projected light andthe second projected light; determining first depth information based onan amount of time for the first projected light to be reflected back tothe receiver when the apparatus operates in the first mode; determiningsecond depth information based on distortion of at least one pattern ofthe second projected light by one or more objects in a scene as detectedby the receiver when the apparatus operates in the second mode; andcompensate for multipath interference (MPI) errors in the first depthinformation at least in part by using the second depth information as areference for removing the MPI errors from the first depth information.